Imaging systems and methods

ABSTRACT

A goggle system is provided. The goggle system includes a computing device, a goggle device configured to be worn by a user and including a detector configured to simultaneously acquire image data of a subject in a first image mode and a second image mode, at least one eye assembly configured to display at least one of an image in the first image mode, an image in the second image mode, and a hybrid image including pixels of image data from the first image mode and pixels of image data from the second image mode, and a communications module configured to transmit acquired image data from the goggle device to the computing device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/689,339 filed Mar. 8, 2022, now allowed, which is a continuation ofU.S. patent application Ser. No. 17/122,848 filed Dec. 15, 2020, nowU.S. Pat. No. 11,310,485, which is a continuation of U.S. patentapplication Ser. No. 16/169,071 filed Oct. 24, 2018, now U.S. Pat. No.10,904,518, which is a continuation of U.S. patent application Ser. No.14/374,002 filed Jul. 23, 2014, now U.S. Pat. No. 10,230,943, which is aNational Stage Entry of PCT/US2013/022704 filed Jan. 23, 2013, whichclaims priority from U.S. Provisional Application No. 61/589,623 filedJan. 23, 2012, all of which are incorporated herein in their entireties.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under grant numbersEB008111 and EB008458 awarded by the National Institutes of Health. Thegovernment has certain rights in the invention.

BACKGROUND

The field of the disclosure generally relates to imaging systems andmethods, and more specifically, to goggles for displaying a plurality ofimage modes.

For surgical operations, in the absence of an image guidance system,surgeons typically need to remove large surgical margins around whatthey perceive as a neoplastic tissue because of the similarity ofdiseased tissue to surrounding healthy tissue. In some parts of thebody, such as the brain, surgeons do not have the luxury of removingsizeable healthy tissue for fear of inducing irreparable damage. Despiteprogress made to enhance the contrast between neoplastic and normaltissues, the human eye is not capable of detecting the contrast signalswith high sensitivity in the operating room. This limitation exasperatestumor resection, resulting in the presence of cancerous cells at or nearthe boundaries of surgically removed tissues.

Recent advances in medical imaging technologies, however, havefacilitated the use of imaging instruments to guide surgical resections.However, at least some known imaging systems have relatively lowsensitivity and accuracy. Further, at least some known imaging systemsgenerally require high costs, complex instrumentation, andtime-consuming image analysis. Moreover, some surgical procedures oftenrequire a team of experienced oncologic surgeons, radiologists, andpathologists to work together. Finally, at least some known imagingsystems include a graphic display of results on a monitor, which candistract surgeons from focusing on the surgery at hand.

SUMMARY

In one aspect, a goggle system is provided. The goggle system includes acomputing device, a goggle device configured to be worn by a user andincluding a detector configured to simultaneously acquire image data ofa subject in a first image mode and a second image mode, at least oneeye assembly configured to display at least one of an image in the firstimage mode, an image in the second image mode, and a hybrid imageincluding pixels of image data from the first image mode and pixels ofimage data from the second image mode, and a communications moduleconfigured to transmit acquired image data from the goggle device to thecomputing device.

In another aspect, a goggle device configured to be worn by a user isprovided. The goggle device includes a detector configured tosimultaneously acquire image data of a subject in a first image mode anda second image mode, and at least one eye assembly configured to displayat least one of an image in the first image mode, an image in the secondimage mode, and a hybrid image including pixels of image data from thefirst image mode and pixels of image data from the second image mode.

In yet another aspect, a method for assembling a goggle device isprovided. The method includes providing a detector configured tosimultaneously acquire image data of a subject in a first image mode anda second image mode, coupling at least one eye assembly to the detectorsuch that the at least one eye assembly is configured to display atleast one of an image in the first image mode, an image in the secondimage mode, and a hybrid image including pixels of image data from thefirst image mode and pixels of image data from the second image mode,and coupling a fastening device to the at least one eye assembly, thefastening device configured to secure the goggle device to a user.

In yet another aspect, a goggle device for fluorescence imagingconfigured to be worn by a user is provided. The goggle device includesa head-mounted display configured to switch between optical see-throughand video see-through modes, a complementary metal-oxide-semiconductor(CMOS) imaging sensor, and a control module configured to interfacebetween the CMOS imaging sensor and a computing device.

In yet another aspect, a method for fluorescence imaging is provided.The method includes administering a fluorescent molecular probe to asubject, observing the subject using a goggle device configured for atleast one of visible light and near infrared light imaging, andidentifying, using the goggle device, at least one tumor in the subjectbased on binding of the fluorescent molecular probe to tissue.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments described herein may be better understood by referringto the following description in conjunction with the accompanyingdrawings.

FIG. 1 is a schematic diagram of an exemplary goggle system.

FIG. 2 is a block diagram of the goggle device shown in FIG. 1 .

FIG. 3 is an image of an alternative goggle device.

FIG. 4 is a schematic diagram of an exemplary detector element that maybe used with the goggle device shown in FIG. 1 .

FIG. 5 is a schematic diagram of a portion of an exemplary detectorarray that may be used with the goggle device shown in FIG. 1 .

FIG. 6 is a schematic diagram of a portion of an exemplary pixel arraythat may be used with the goggle device shown in FIG. 1 .

FIGS. 7A-7C are schematic depictions of images shown on the goggledevice shown in FIG. 1 .

FIGS. 8A-8C are images shown on the display of the goggle device shownin FIG. 1 .

FIG. 9 is an image of an alternative goggle device.

FIG. 10 is a block diagram of the goggle device shown in FIG. 9 .

FIG. 11 is a schematic diagram of a display module of an alternativegoggle device.

FIG. 12 is a schematic diagram of an imaging and illumination modulethat may be used with the alternative goggle device shown in FIG. 11 .

FIG. 13 is a schematic diagram of a filter array that may be used withthe imaging and illumination module shown in FIG. 12 .

FIGS. 14A-14D are exemplary fluorescent molecular probes.

FIG. 15 is a schematic diagram of an exemplary synthesis for an LS-301fluorescent molecular probe.

DETAILED DESCRIPTION

Embodiments described herein provide a goggle system that includes agoggle device in communication with a computing device. The goggledevice enables a user to view a subject in a plurality of imaging modesin real-time. The imaging modes include a hybrid-imaging mode thatsimultaneously displays pixels of image data of a first imaging mode andpixels of image data of a second imaging mode. Accordingly, a user isable to quickly and easily visualize a subject during a surgicaloperation.

FIG. 1 is a schematic diagram of an exemplary goggle system 100. FIG. 2is a block diagram of a goggle device 102 that may be used with gogglesystem 100. Goggle device 102 may be worn by a user 104, such as asurgeon, and aids user 104 in viewing a subject 106, as described indetail herein. In the exemplary embodiment, goggle device 102 transmitsand/or receives data from a computing device 108. Data may betransferred between computing device 108 and goggle device 102 over awired or wireless network.

In the exemplary embodiment, goggle device 102 includes a left eyeassembly 110 that displays an image for a left eye of user 104, and aright eye assembly 112 that displays an image for a right eye of user104. Goggle device 102 further includes a fastening device 114, such asa strap, for securing goggle device 102 to user 104 during use.

Goggle device 102 enables user 104 to view subject 106 in a plurality ofimage modes. In the exemplary embodiment, goggle device 102 enables user104 to view a far- or near-infrared (NIR) image of subject 106, avisible light image of subject 106, and a hybrid near-infrared/visiblelight image of subject 106. Alternatively, goggle device 102 may enableuser 104 to view a subject in any image mode and/or combination of imagemodes, including, for example, photoacoustic images, interferenceimages, optical coherence tomography images, diffusion opticaltomography images, polarization images, ultrasound images, magneticresonance imaging (MRI) images, nuclear images (e.g., positron emissiontomography (PET) images, single-photon emission computed tomography(SPECT) images), computed tomography (CT) images, gamma-imaging, andX-ray images.

A switch 116 on goggle device 102 enables user 104 to switch betweenimage modes. In the exemplary embodiment, switch 116 is a toggle switch.Alternatively, switch 116 may be any type of switching device thatenables goggle device 102 to function as described herein. For example,in one embodiment, switch 116 is a foot pedal, and user 104 activatesswitch 116 by depressing the foot pedal. In another embodiment, switch116 is voice-activated.

Switch 116 controls which image mode is displayed on left eye assembly110 and right eye assembly 112. In the exemplary embodiment, switch 116controls whether left eye assembly 110 and/or right eye 112 assemblydisplays a visible light image, a near-infrared image, or a hybridnear-infrared/visible light image. In one embodiment, left eye assembly110 and right eye assembly 112 each have a respective switch 116.Accordingly, in some embodiments, user 104 may view a first image modeon left eye assembly 110 and a second image mode on right eye assembly112.

Goggle device 102 further includes sources for imaging. In oneembodiment, the goggle device includes a white light (i.e., visiblelight) source 120 and a near-infrared light source 122. White lightsource 120 and near-infrared light source 122 illuminate subject 106with visible light and near-infrared light, respectively. In otherembodiments, different light sources may be utilized for differentspecific imaging modes. For example, suitable light sources may beintegrated with goggle device 102 to enable the capture of photoacousticimages, interference images, optical coherence tomography images,diffusion optical tomography images, polarization images, far infraredimages, thermal images, ultrasound images, and nuclear images (e.g.,PET, SPECT, CT, gamma-imaging, X-ray).

In the exemplary embodiment, white light source 120 includes a pluralityof white light-emitting diodes (LEDs), and near-infrared light source122 includes a plurality of near-infrared LEDs for fluorescence imaging.White light source 120 and near-infrared light source 122 each provide afield of view of approximately 0.3 m in diameter at 1.0 m from therespective source. Alternatively, white light source 120 andnear-infrared light source 122, including any components that enablegoggle device 102 to function as described herein, such as, for example,laser, laser diodes, light bulbs, or any combination of theaforementioned components. Alternatively, near-infrared light source 122may include light sources in other wavelengths to enable absorption orluminescence, or fluorescence imaging in other spectral windows.

Near-infrared light emitted from near-infrared light source 122 excitesfluorescent molecular probes present in subject 106. For example, for atumor resection operation, a molecular probe capable of fluorescentexcitation is injected into a subject. The molecular probe includes apeptide sequence and a near-infrared fluorescent dye, such asindocyanine green dye, having absorption and emission maxima around 780nm and 830 nm, respectively. After injecting the molecular probe intothe subject 106, the molecular probe binds to tumors. Accordingly, whennear-infrared light from near-infrared light source 122 strikes subject106, the fluorescent dye in the molecular probe is excited. With thefluorescent dye excited, lesions such as tumors can quickly and easilybe visualized in the near-infrared imaging mode of goggle device 102. Asimilar procedure is applicable in other spectral regions.

To transfer data between goggle device 102 and one or more remotedevices, such as computing device 108, the goggle device includes acommunications module 130. In the exemplary embodiment, for transfer ofdata over a wireless network, communications module 130 includes aradio-frequency (RF) transmitter/receiver. Data transfer can also beaccomplished through other suitable platforms such as, for example,Bluetooth, WI-FI, infrared (IR) communication, internet, 3G, 4G network,satellite, etc. Communications module 130 may also be transfer data tocomputing device 108 over a wired connection, such as for example, a USBvideo capture cable. Communications module 130 enables image datacollected by goggles to be displayed and/or stored on computing device108. Accordingly, image data acquired by goggle device 102 can be viewednot only on goggle device 102 but also on computing device 108.

Goggle device 102 includes a power module 140 that supplies power togoggle device 102. In the exemplary embodiment, power module 140 is abattery unit that stores and provides electrical energy to goggle device102. Alternatively, power module 140 is any device configured to supplypower to goggle device 102.

To acquire image data of subject 106 to display on left eye assembly 110and right eye assembly 112, goggle device 102 includes a detector module150. In the exemplary embodiment, detector module 150 is a hybriddetector array capable of detecting both near-infrared and visiblelight. Detector module 150 is mounted on the front of goggle device 102to collect image data from a direction that user 104 is facing. Detectormodule 150 displays received image data on left eye assembly 110 andright eye assembly 112 such that left eye assembly 110 and right eyeassembly 112 display the same regions of subject 106 that a left eye andright eye of user 104 would observe in the absence of goggle device 102.

FIG. 3 is an image of an alternative goggle device 300 that may be usedwith goggle system 100 (shown in FIG. 1 ). Goggle device 300 includescomponents substantially similar to goggle device 102, and likereference numerals are used herein to identify like components. Unlikegoggle device 102, goggle device 300 includes only a single eye assembly302. As such, when using goggle device 300, one eye of user 104 viewssubject 106 through single eye assembly 302, and the other eye of user104 views subject uninhibited (i.e., with the naked eye).

Single eye assembly 302 functions substantially similar to left eyeassembly 110 and right eye assembly 112 (shown in FIGS. 1 and 2 ). Inthe exemplary embodiment, single eye assembly 302 covers the left eye ofuser 104. Alternatively, single eye assembly 302 may cover the right eyeof user 104.

FIG. 4 is a schematic diagram of a detector element 400 that may be usedwith detector module 150 to collect image data for goggle device 102(both shown in FIGS. 1 and 2 ). Detector element 400 includes aplurality of differential layers 402 (denoted by Dn) along adifferential length in a continuum. In the exemplary embodiment, eachdifferential layer 402 is a p-n junction 404. In the exemplaryembodiment, detector element 400 includes eleven differential layers402. Alternatively, detector element 400 may include any number ofdifferential layers 402 that enables detector element 400 to function asdescribed herein. The longer the wavelength of the incident light, thedeeper the light will penetrate detector element 400. Accordingly, eachlayer 402 is configured to detect a different frequency range ofincident light. For example, in one embodiment, channels D5-D8 detectblue light (i.e., light with a wavelength of approximately 440-490 nm),and the total wavelength spectrum detectable by detector element 400 isless than 300 nm to greater than 1300 nm. By using a continuum ofdifferential layers 402, detector element 400 is capable ofsimultaneously detecting a broad range of wavelengths of the incidentlight, including visible light (i.e., red, green, and blue light),near-infrared light, and various other wavelengths.

Images can be generated from the image data acquired by a selection ofdifferential layers 402. For example, to generate a visible light image,image data from differential layers 402 corresponding to red, blue, andgreen light is used. Further, images may be generated using addition,subtraction, integration, differentiation, and/or thresholding ofdifferential layers 402. Accordingly, image data acquired using detectorelement 400 may be used to generate a variety of images.

FIG. 5 is a schematic diagram of a portion of a detector array 500 thatmay be used with detector module 150 (shown in FIG. 2 ). Detector array500 includes a plurality of detector elements 502. In some embodiments,each detector element 502 corresponds to one pixel. In the exemplaryembodiment, array 500 includes near-infrared detector elements 504 andvisible light detector elements 506 arranged in a checkerboard pattern.That is, near-infrared detector elements 504 and visible light detectorelements 506 alternate in both a horizontal and vertical direction.Notably, this concept is not restricted to near-infrared detectorelements or visible pixel elements. That is, for different imagingapplications, near-infrared detector elements 504 and/or visible pixeldetector elements 506 may be replaced detector elements in otherwavelength windows, such as, for example, 500 nm-550 nm, 600 nm-630 nm,1100 nm-1150 nm, 1200 nm-1300 nm, etc.

In the exemplary embodiment, visible light detector elements 506 arecontinuum detector elements, such as detector element 400 (shown in FIG.4 ), and near-infrared detector elements 504 are single-channel detectorelements. Alternatively, visible light detector elements 506 andnear-infrared detector elements 504 may be any type of detector elementthat enables detector array 500 to function as described herein.

FIG. 6 is a schematic diagram of a portion of a pixel array 600displayed by left eye assembly 110 and right eye assembly 112. Similarto the arrangement of detector array 500, the pixel array 600 includes aplurality of pixels 602, including near-infrared pixels 604 and visiblelight pixels 606 arranged in a checkerboard pattern. In the exemplaryembodiment, detector elements 502 correspond to pixels 602. As such, inthe exemplary embodiment, to display different imaging modes on left eyeassembly 110 and/or right eye assembly 112, different combinations ofdetector elements 602 are utilized such that different combinations ofnear-infrared pixels 604 and visible light pixels 606 and are displayed.

For example, when switch 116 (shown in FIG. 1 ) is set to a visiblelight imaging mode, visible light pixels 506 are used, and near-infraredpixels 504 are not used for imaging, such that visible light pixels 606create a visible-light image. When switch 116 is set to a near-infraredimaging mode, near-infrared pixels 504 are used, and visible lightpixels 506 are not used for imaging, such that near-infrared pixels 604create a near-infrared image. Finally, in a hybrid imaging mode, bothnear-infrared detectors 504 and visible light detectors 506 are used forimaging, creating an image including near-infrared pixels 604 andvisible light pixels 606. Filtering for the different imaging modes maybe accomplished by a filter and/or polarizer (neither shown).

FIGS. 7A-7C are schematic diagrams of a near-infrared image 702, avisible light image 704, and a hybrid image 706, respectively, that maybe displayed on goggle device 102 (shown in FIGS. 1 and 2 ). Innear-infrared image 702, the molecular probe excited by near-infraredlight source 122 (shown in FIG. 1 ) clearly delineates a tumor 708, butthe rest of a subject 710 is not visible. In visible light image 704,subject 710 is visible, but tumor 708 is not visible. In hybrid image706, however, both tumor 708 and subject 710 are visible. Accordingly,when user 104 views hybrid image 706 on goggle device 102, both tumors708 and subject 710 are visible, enabling user 104 to better perform asurgical operation.

FIGS. 8A-8C are a near-infrared image 802, a visible light image 804,and a hybrid image 806, respectively, of a mouse 808 with tumors 810. Innear-infrared image 802, the tumors 810 are visible, but the rest ofmouse 808 is not visible. In visible light image 804, mouse 808 isvisible, but tumors 810 are not visible. In hybrid image 806, however,both tumors 810 and mouse 808 are visible. Notably, using communicationsmodule 130 (shown in FIG. 2 ), images 802, 804, and/or 806 may also beviewed on computing device 108 (shown in FIG. 1 ).

FIG. 9 is an image of an alternative goggle device 900 that may be usedwith goggle system 100 (shown in FIG. 1 ). FIG. 10 is a block diagram ofthe goggle device 900 shown in FIG. 9 . In the exemplary embodiment,goggle device 900 is a head-mounted display (HMD) capable of fasttemporal resolution and switching between an optical see-through modeand a video see-through mode. In the optical see-through mode, user 104sees the real world through half-transparent mirrors, and thehalf-transparent mirrors are also used to reflect computer-generatedimages into the eyes of user 104, combining real and virtual worldviews. In the video see-through mode, cameras (or other suitabledetection devices) capture the real-world view, and computer-generatedimages are electronically combined with the video representation of thereal world view. As both the real and virtual world images are digitalin the video see-through mode, lag between the real and virtual worldimages can be reduced. In the optical see-through mode, user 104 canvisualize surroundings with natural vision. In the video see-throughmode, real-time NIR fluorescence video is presented to user 104 withrelatively high contrast. Allowing user 104 to switch between opticaland video see-through modes simplifies surgical operations and allowsuser 104 to visualize subject 106 with natural vision or enhanced visionas desired.

Goggle device 900 includes a complementary metal-oxide-semiconductor(CMOS) imaging sensor 902 integrated onto a custom printed circuit board(PCB) platform (not shown). Alternatively, goggle device 900 may includea charge-coupled device (CCD) imaging sensor. A long-pass filter 904 ismounted on an objective imaging lens 906. In the exemplary embodiment,long-pass filter 904 is an 830 nanometer (nm) filter.

A control module 908 interfaces between CMOS imaging sensor 902 and acomputing device 910, such as computing device 108. In the exemplaryembodiment, control module 908 includes a field-programmable gate array(FPGA) integration model with a universal serial bus (USB) communicationcapabilities and a laptop computer. Data received by CMOS imaging sensor902 is read out in multiple stages. In the exemplary embodiment, datafrom CMOS imaging sensor 902 is read out via a state machine implementedon the FPGA, and the data is stored in a first in first out (FIFO)process and transferred to a first synchronous dynamic random-accessmemory (SDRAM) chip in control module 908. In the exemplary embodiment,control module 908 includes two SDRAM chips, such that a first SDRAMchip stores pixel data from CMOS imaging sensor 902, while a secondSDRAM chip transfers the data to an output FIFO on the FPGA fortransferring the data to computing device 910 via a universal serial bus(USB). In some embodiments, control module 908 may include a datacompression chip for compressing the data.

To display information to user 104, goggle device 900 includes an HMDunit 920 that interfaces with computing device 910 via a high-definitionmultimedia interface (HDMI) link to display real-time images on HMD unit920. Goggle device 900 further includes and NIR light source 922 thatemits NIR light through illumination optics 924 and a short-pass filter926 to illuminate fluorescent molecular probes (such as indocyaninegreen dye) in a surgical field 930. Surgical field 930 may be, forexample, a portion of subject 106 (shown in FIG. 1 ). In the exemplaryembodiment, NIR light source 922 includes four NIR LEDs, and short-passfilter 926 is a 775 nm filter.

A sensitivity of goggle device 900 to detect a fluorescence signal fromsurgical field 930 is characterized using a signal-to-noise ratio (SNR),which compares a level of the desired signal relative to a noise level.Pixel binning and temporal averaging may be used to improve SNR ofgoggle device 900. Pixel binning involves combining signals from a groupof pixels in a spatial domain, which is analogous to increasing thenumber of photons that contribute to the detected signal Binningimproves the SNR estimate by the square root of the number of pixelsbinned. For instance, binning a neighborhood of 2 by 2 pixels improvesthe SNR by a factor of 2. However, improvement in SNR due to binningoccurs at the expense of reduced spatial resolution and loss of highspatial frequency components in a final image.

Temporal averaging involves combing signals from a group of pixels in atime domain, which, like binning, is also analogous to increasing thenumber of photons that contribute to the detected signal. Temporalaveraging increases SNR by the square root of the number of averagedpixels in the time domain. Hence, temporal averaging of four consecutiveframes will increase SNR by a factor of 2. However, temporal averagingmay create image lag when a moving target is imaged.

Both temporal averaging and pixel binning may be combined together tofurther improve SNR. For example, averaging 4 frames as well asaveraging a pixel neighborhood of 2 by 2 pixels will improve SNR by afactor of 4 while reducing spatial and temporal resolution by a factorof 4. It was determined experimentally that SNR increases linearly withexposure time at a rate that depends on the concentration of afluorescent molecular probe (e.g., indocyanine green dye). As theexposure time increases, SNR increases at the cost of a slower framerate. Using goggle device 900 experimentally, sentinel lymph nodemapping was performed on rats using NIR quantum dots (QDs), andfluorescence-guided liver surgery and intraoperative imaging wereperformed on mice. Goggle device 900 is capable of real-timefluorescence imaging of up to 60 frames per second (fps).Experimentally, it was determined that goggle device 900 detectsfluorescence signals as low as 300 picomolar (pM) of indocyanine greendye. Compared to a charge-coupled device (CCD) imaging sensor, which has20% quantum efficiency at 830 nm, CMOS imaging sensor 902 has a quantumefficiency of greater than 30% at 830 nm. Further, in the exemplaryembodiment, goggle device 900 includes one or more buttons and/orswitches (neither shown) for user 104 to select automatic or manual gainand automatic or manual exposure time.

FIG. 11 is a schematic diagram of a display module 1101 for analternative goggle device 1100 that may be used with goggle system 100(shown in FIG. 1 ). Similar to goggle device 900, goggle device 1100 isimplemented in a dual-mode visual and optical see-through HMD. However,unlike goggle device 900, goggle device 1100 provides three-dimensional(3D) imaging and display, as described herein. In one embodiment, afield of view for illumination and imaging of goggle device 1100 is 300mm×240 mm at a distance between goggle device 1100 and subject 106 of0.3 m-1.2 m.

Display module 1101 of goggle device 1100 includes a first organiclight-emitting diode (OLED) 1102 and a second OLED 1104 to displayimages to a right eye 1106 and left eye 1108, respectively of user 104.First OLED 1102 emits light through a first optical assembly (e.g.,imaging and/or focusing lenses) 1110, and second OLED 1104 emits lightthrough a second optical assembly 1112.

OLED display technology provides benefits over liquid crystal display(LCD) technology as it uses approximately 80% less power than LCDs, hasa nominal viewing area of approximately 160° (approximately 265% largerthan LCDs), a nominal contrast ration of (as compared to 60:1 for LCDs),and a significantly faster refresh rate, reducing eye fatigue andheadaches. The OLED microdisplay is also more compact than its LCDcounterpart because no additional illumination is needed. The proposedfield of view of the display is 45°×36° with a microdisplay resolutionof 1280×1024 (SXGA). In the exemplary embodiment, the pupil size of thegoggle device 1100 is 10 mm in diameter. Off-axis design with asphericalplastic elements may be used to reduce the size and weight of goggledevice 1100.

To switch between optical and video see-through modes, goggle device1100 includes a fast liquid crystal shutter 1120 positioned in front ofa combiner 1122. In the exemplary embodiment, combiner 1122 is a plasticelement with 50% reflection on an inner (i.e., eye-facing) surface suchthat user 104 can see through combiner 1122 in optical see-through mode,and information from OLEDs 1102 and 1104 can be directed to user 104 inboth modes.

When an external voltage is applied, fast liquid crystal shutter 1120 istransparent and transmits light. Without an external voltage, fastliquid crystal shutter 1120 blocks light from the surgical field andenvironment. Therefore, the goggle device 1110 can be switched betweenoptical and video see-through modes easily and rapidly. In someembodiments, a switch (not shown) may be controlled by a foot paddle toenable hands-free operation. Using the video-see-through mode of goggledevice 1100, 3D reflectance images and fluorescence images can beregistered and presented precisely. The 3D fluorescence images can alsobe viewed with the optical-see-through mode, while user 104 views thesurrounding environment naturally.

FIG. 12 is a schematic diagram of an imaging and illumination module1150 for goggle device 1100. For true stereoscopic vision, imaging andillumination module 1150 includes two separate and identical imagingsystems and one illumination system between and above the imagingsystems that provide uniform NIR illumination to excite fluorescentmolecular probes and visible light illumination for reflectance imaging.Each of the two imaging systems includes a CMOS detector 1152 and animaging lens 1154 in the exemplary embodiment. CMOS detectors 1152provide higher resolution and faster frame rates than CCD detectors. Inthe exemplary embodiment, the distance between the imaging systems is 67mm (the average inter-pupillary distance for adults).

As shown in FIG. 12 , to provide a well-defined and uniform illuminationregion, the illumination system includes a first LED array 1160 and asecond LED array 1162. In the exemplary embodiment, first LED array 1160is a high power 780 nm LED array that includes 60 diode chips andoptical output power of 4 Watts (W), and second LED array 1162 is a highpower white light LED array that includes 16 diode chips to provideuniform illumination over an area of 300×240 mm. The power of NIRexcitation is approximately 3 mW/cm 2 in the exemplary embodiment.

Light from first and second LED arrays 1160 and 1162 is combined using adichroic mirror 1170 such that the illumination area from both first andsecond LED arrays 1160 and 1162 overlaps. The combined light isdistributed using illumination optics 1172. Illumination optics 1172, inthe exemplary embodiment, include freeform plastic lenses (not shown)that generate uniform light distribution and an excitation filter (notshown) that blocks excitation light over 800 nm.

Each CMOS detector 1152 and imaging lens 1154 capture white lightreflectance and NIR fluorescence images simultaneously. In the exemplaryembodiment, each CMOS detector 1152 includes a sensor ofvertically-stacked photodiodes and pixelated NIR/visible spectrumfilters. More specifically, in the exemplary embodiment, CMOS detector1152 includes an array of 2268×1512 imaging pixels, and each pixelincludes three vertically-stacked photodiodes that can separate spectraof blue, green, and red-NIR light for color imaging. As each pixelincludes three vertically-stacked diodes, unlike at least some knownimaging systems, there is no need to interpolate between neighboringpixels. Experimentation suggests that the quantum efficiency of thevertically-stacked photodiode sensor is approximately 35% at 800 nm,which is significantly better than at least some known CCD sensors andideal for NIR imaging. The scanning rate of CMOS detector 1152 may be asfast as 40 fps, and a subset of the pixel array can be read out athigher frame rates (e.g., 550 fps for 128×128 pixels).

FIG. 13 is a schematic diagram of a filter array 1300 that may be usedwith CMOS detector 1152 (shown in FIG. 12 ). Filter array 1300alternates between pixels with a visible spectrum filter 1302 and pixelswith a NIR filter 1304. Filter array 1300 may be created by taking a NIRfilter the size of the entire array 1300 and selectively removingportions using a focus ion beam (FIB). Visible filters 1302, which arethen selectively removed parts of the larger NIR filter, will allowpassage of visible spectrum light, which will be subsequently absorbedby the vertically-stacked photodiodes 1306 for color separation.

Thus, white light reflectance imaging can be achieved with visiblepixels. On the other hand, NIR filters 1304 will only allow NIR light ofinterest (λ>820 nm) to pass, and the NIR signal will be read out fromthe deepest of vertically-stacked photodiodes 1306. Due to the neteffect of both spectrum separation mechanisms (NIR filter 1304 andvertically-stacked photodiodes 1306), the performance of NIRfluorescence detection will be further enhanced compared to theconventional method of using a NIR filter alone. The CMOS detector 1152enables co-registration of color images and NIR images on-chip whilereducing the number of sensors required for 3D reflectance/fluorescenceimaging. This facilitates eliminating artifacts in co-registration dueto motion and minimizes the delay due to exhausting off-chipcomputation.

Goggle device 1100 includes an autofocus feature without moving partsthat optimizes a focus in the exemplary embodiment. A zoom lens with acompact piezo actuator or a liquid lens with a variable,voltage-dependent focal length may be used to implement the autofocusfeature. In the exemplary embodiment, image processing for goggle device1100 is performed by an FPGA coupled to CMOS detectors 1152 and OLEDs1102 and 1104.

The goggle devices described herein may be used with or without contrastagents. In the absence of extrinsic contrast agents, imaging signals forendogenous fluorophores or biomolecules may be used to provide imagingcontrast. At least some embodiments utilize NIR fluorescent orluminescent molecules or materials that localize selectively in a tissueof interest. As noted above, indocyanine green dye may be used as afluorescent molecular probe with the goggle devices described herein.Other fluorescent dyes such as NIR pyrrolopyrrole cyanine dyes orluminescent materials such as quantum dots or dye-loaded nanoparticlesmay be utilized. However, uptake of high-affinity probes in small tumorcells may be overwhelmed by background fluorescence from normal tissue,decreasing contrast between tumor cells and background tissue. Instead,the fluorescent molecules such as dyes, or materials such as luminescentnanoparticles, could be linked to another molecule or group of moleculesthat will improve selective uptake in the tissues or cells of interest.For example, fluorescent molecular probes that bind selectively toprotein receptors or other biomarkers overexpressed in tumor cells ortarget tissue may also be utilized with the goggle devices describedherein.

FIGS. 14A-14D are a plurality of exemplary fluorescent molecular probes.FIG. 14A shows indocyanine green dye. Cypate, shown in FIG. 14B, is aNIR cyanine dye with similar spectral properties to indocyanine greendye. Cypate, LS-276, shown in FIG. 14C, and LS-288, shown in FIG. 14D,are models of hydrophobic, intermediate hydrophilic, and hydrophilicdyes, respectively.

An example of a tumor-targeted molecular probe is LS-301, which has thestructure, Cypate-cyclo (D-Cys-Gly-Arg-Asp-Ser-Pro-Cys)-Lys-OH. Thespectral properties of LS-301 are suitable for NIR imaging applications(e.g., excitation/emission 790/810 nm in 20% DMSO solution; fluorescencequantum yield (ψ) 0.1 referenced to ICG). The ABIR binding affinity forLS-301 is Ki=26.2±0.4 nM relative to reference cypate-labeled RGDpeptide (Ki=1.2±0.7 nM). In addition to conferring tumor selectivity onLS-301, the unnatural D-cysteine on the peptide moiety confers higherstability because of its resistance to degradation by proteases. Thisbiological stability in serum allows initial imaging and potentialfollow-up surgery to be conducted within 48 hours before subsequenthydrolysis of the probe through L-amino acid residues.

Experimentally, NIR fluorescence microscopy of LS-301 in diverse tumorcells showed punctate intracellular fluorescence typical ofreceptor-mediated endocytosis and barely visible uptake in non-tumorcells. This uptake was successfully inhibited with unlabeled cyclic(RGDFV) reference peptide in A549 tumor cells, demonstrating theversatility of the imaging probe in detecting tumors relative tonon-tumor cells.

Hydrophobic dyes, such as cypate, bind to albumin and other proteins.The high binding constant decreases their bioavailability for the targettumors and prolongs the blood circulation time, thereby increasingbackground fluorescence at early imaging time points. In contrast, morehydrophilic dyes and their peptide conjugates rapidly extravasate intotissues, quickly removing the probe from circulation. Although thehydrophilic probes are suitable for image-guided surgery because of thefast clearance, the contrast between tumor and surrounding tissue alsodepends on having sufficient time for molecular interaction between thetarget tumor proteins and the molecular probe. Thus, the low backgroundsignal obtained may be offset by the low signal at the tumor site.Experimental data suggest that LS-276 dye (shown in FIG. 14C) willbridge the gap between rapid and delayed blood clearance, which affectsthe bioavailability of the probes to a target tissue.

Due to a direct linkage of a carboxylic acid with a phenyl group inLS-276, LS-276 may have relatively low reactivity with peptides andproteins, resulting in multiple side products that are difficult toremove. Accordingly, in some embodiments, a fluorophore based on abenzyl instead of the current phenyl carboxylic acid used for LS-276 maybe utilized. Since the pure compound is a solid, re-crystallizationmethods may be used where ethyl acetate and chloroform mixtures are usedto precipitate the dye in >99% HPLC/HRMS purity. FIG. 15 is a schematicdiagram of the synthesis of reactive benzylcarboxylic acid forsolid-phase labeling of LS-301 peptide. Using this derivative of LS-276may double the quantum yield of cypate used in LS-301. In someembodiments, the method may produce the desired compound in high yield(>70%) and purity (>99%). The method is also scalable, with thepotential to produce up to 10 grams of compound.

In some embodiments, the LS-301 peptide may be slightly altered toassess further improvements in tumor selectivity (e.g.,cyclo(DCys-Gly-Arg-Asp-Ser-Pro-DCys)-Lys-OH,cyclo(Cys-Gly-Arg-Asp-Ser-Pro-Cys)-Lys-OH,cyclo(Cys-Arg-Gly-Asp-Ser-Pro-Cys)-Lys-OH,cyclo(DCys-Arg-Gly-Asp-Ser-Pro-Cys)-Lys-OH, andcyclo(DCys-Arg-Gly-Asp-Ser-Pro-DCys)-Lys-OH). These peptides are labeledwith dye 2 at the N-terminus.

The goggle devices and fluorescent probes described herein may beimplemented in a plurality of surgical settings, including, but notlimited to, detecting tumors related to breast cancer in mice,adenocarcinoma in canines, and hepatocellular carcinoma (HCC) in humans.

The goggle devices and fluorescent probes described herein assist inidentifying tumor boundaries and performing biopsies. The goggle devicesand fluorescent probes described herein are also applicable to othersurgical interventions such as cardiac surgery and assessing woundhealing.

In one example, goggle device 102 (shown in FIG. 1 ) and the fluorescentprobe LS-301 described herein were utilized in mapping positive lymphnodes (PLNs) in mice. Based on the information provided by non-invasivefluorescence imaging using goggle device 102 and LS-301, regions ofinterest that might contain PLNs were explored and examined PLNs wereidentified and resected under image guidance. The presence of canceroustissue in the PLNs was confirmed by bioluminescence imaging. Thisverifies the feasibility of the non-invasive fluorescence imaging,allowing clinicians to rapidly stage cancer non-invasively. Using goggledevice 102 and fluorescent probe LS-301 non-invasively can provide afirst-line screening that provides general information in an operatingroom in real-time. Further, the fluorescence signals monitored in PLNsmight be used as an indicator of the efficacy of treatment regimens suchas radiotherapy, chemotherapy, and targeted therapy.

In another example, a multimodal detection technique was implemented inwhich goggle-aided fluorescence imaging (e.g., using goggle device 102(shown in FIG. 1 ) and indocyanine green dye) was combined withultrasound imaging and standard histology. Specifically, fluorescenceimaging was used to detect liver tumors in mice and to perform liverresections on the tumors. In addition to single tumors, scatteredsatellite lesions were also detected. Further, liver resection wasperformed on a rabbit using ultrasound, fluorescence imaging, andstandard histology. The presence of tumors in the rabbit was confirmedby ultrasound and then observed in real-time using fluorescence gogglesystem 100 (shown in FIG. 1 ). The excised tissues were later examinedby histopathology, and it was confirmed that the tumors were cancerous.

In another example, goggle device 102 (shown in FIG. 1 ) and indocyaninegreen dye were used to image hepatocellular carcinoma (HCC) in humanpatients. Both intravenous and transarterial hepatic (TAH) delivery ofindocyanine green dye were used. Primary tumors and satellite tumorswere both detected using fluorescence imaging, some of which were notidentified in pre-operative MRI and CT images or by visual inspectionand palpation. Histologic validation was used to confirm HCC in thepatients. The HCC-to-liver fluorescence contrast detected by goggledevice 102 was significantly higher in patients that received TAHdelivery instead of intravenous delivery.

The systems and methods described herein provide a goggle device incommunication with a computing device. The goggle device enables a userto view a subject in a plurality of imaging modes in real-time. Theimaging modes include a hybrid-imaging mode that simultaneously capturesand displays pixels of image data of a first imaging mode and pixels ofimage data of a second imaging mode. Accordingly, a user is able toquickly and easily visualize a subject during a surgical operation.

Notably, the goggle system and goggle device described herein may beutilized in a broad variety of medical applications, including smallanimal imaging, veterinary medicine, human clinical applications,endoscopic applications, laparoscopic applications, dental applications,cardiovascular imaging, imaging inflammations, wound healing, etc.Further, the goggle system and goggle device described herein may beused in other imaging applications outside of the medical field.

The order of execution or performance of the operations in theembodiments of the disclosure illustrated and described herein is notessential unless otherwise specified. That is, the operations may beperformed in any order, unless otherwise specified, and embodiments ofthe disclosure may include additional or fewer operations than thosedisclosed herein. For example, it is contemplated that executing orperforming a particular operation before, contemporaneously with, orafter another operation is within the scope of aspects of thedisclosure.

When introducing elements of aspects of the disclosure or embodimentsthereof, the articles “a,” “an,” “the,” and “said” are intended to meanthat there are one or more of the elements. The terms “comprising,”including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.

This written description uses examples to disclose the disclosure,including the best mode, and also to enable any person skilled in theart to practice the disclosure, including making and using any devicesor systems and performing any incorporated methods. The patentable scopeof the disclosure is defined by the claims and may include otherexamples that occur to those skilled in the art. Such other examples areintended to be within the scope of the claims if they have structuralelements that do not differ from the literal language of the claims orif they include equivalent structural elements with insubstantialdifferences from the literal language of the claims.

1-29. (canceled)
 30. A device, comprising: a detector configured tosimultaneously acquire image data from a plurality of fluorescentmolecules, a plurality of luminescent materials, or a combinationthereof in a state of excitation caused by a fluorescence excitationlight source, and from visible light illuminated by a white lightsource; and at least one display module configured to display pixels ofthe acquired image data in real-time.
 31. The device of claim 30,wherein the white light source comprises a plurality of white lightemitting diodes.
 32. The device of claim 30, further comprising a secondfluorescence excitation light source.
 33. The device of claim 30,wherein the at least one display module is further configured to displayan image chosen from a first image mode associated with the image datafrom the plurality of fluorescent molecules, the plurality ofluminescent materials, or a combination thereof, and a second image modeassociated with the image data from the visible light.
 34. The device ofclaim 30, wherein the at least one display module comprises at least oneeye assembly.
 35. The device of claim 30, further comprising anendoscope.
 36. The device of claim 35 adapted for laparoscopicapplications.
 37. The device of claim 33, wherein the first image modeis a near-infrared image mode.
 38. The device of claim 30, furthercomprising a switch configured to switch between different imaging modesdisplayed in real-time on the at least one display module.
 39. Thedevice of claim 30, wherein the plurality of luminescent materials islocalized in a tissue of interest, and the fluorescence emitted by theexcited plurality of luminescent materials is absorbed by the pluralityof fluorescent molecules.
 40. The device of claim 30, wherein theplurality of fluorescent molecules, the plurality of luminescentmaterials, or the combination thereof comprises a fluorescent molecularprobe chosen from LS-301.
 41. A system comprising: a computing device; adetector configured to simultaneously acquire image data from aplurality of fluorescent molecules, a plurality of luminescentmaterials, or a combination thereof in a state of excitation caused by afluorescence excitation light source, and from visible light illuminatedby a white light source; and at least display module configured todisplay pixels of the acquired image data in real-time.
 42. The systemof claim 41, wherein the white light source comprises a plurality ofwhite light emitting diodes.
 43. The system of claim 41, furthercomprising a second fluorescence excitation light source.
 44. The systemof claim 41, wherein the at least one display module is furtherconfigured to display an image chosen from a first image mode associatedwith the image data from the plurality of fluorescent molecules, theplurality of luminescent materials, or a combination thereof, and asecond image mode associated with the image data from the visible light.45. The system of claim 41, further comprising an endoscope.
 46. Thesystem of claim 45 adapted for laparoscopic applications.
 47. A methodfor detection using a device, the method comprising: acquiring imagedata with a device of claim 30 from a subject, the image data beingacquired from a plurality of fluorescent molecules, a plurality ofluminescent materials, or a combination thereof in a state of excitationcaused by a fluorescence excitation light source, and from visible lightilluminated by a white light source; and displaying pixels of theacquired image data from the subject in real-time on the at least onedisplay module of the device.
 48. The method of claim 47, furthercomprising administering the plurality of fluorescent molecules, theplurality of luminescent materials, or the combination thereof to asubject.
 49. The method of claim 48, wherein the plurality offluorescent molecules, the plurality of luminescent materials, or thecombination thereof comprises a fluorescent molecular probe chosen fromLS-301.